2026-09-21 00:34:26,214 [ 326056 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2026-09-21 00:34:26,214 [ 326056 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2026-09-21 00:34:26,214 [ 326056 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2026-09-21 00:34:26,214 [ 326056 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_nsjvi9 --privileged --dns-search='.' --memory=30703247360 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_OLD_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel2_0.jsonl --report-log-exclude-logs-on-passed-tests 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1]' test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards test_part_uuid/test.py::test_part_uuid test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple test_partition/test.py::test_system_detached_parts test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_relative_filepath/test.py::test_filepath test_reload_client_certificate/test.py::test_correct_cn_cert test_reload_client_certificate/test.py::test_wrong_cn_cert test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_s3_imds/test_session_token.py::test_credentials_from_metadata test_s3_imds/test_simple.py::test_credentials_from_metadata test_s3_latency_log/test.py::test_latency_log test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 test_s3_storage_class/test.py::test_s3_storage_class_right 'test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3]' test_s3_style_link/test.py::test_s3_table_functions test_s3_style_link/test.py::test_s3_table_functions_line_as_string test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_reloading_storage_configuration/test.py::test_add_disk test_partition/test.py::test_attach_check_all_parts test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] test_replication_credentials/test.py::test_credentials_and_no_credentials test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] test_recovery_replica/test.py::test_choose_source_replica test_rocksdb_options/test.py::test_invalid_block_based_table_options [gw3] [ 1%] PASSED test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part [gw3] [ 2%] PASSED test_partition/test.py::test_cannot_attach_active_part [gw9] [ 3%] PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 test_partition/test.py::test_detached_part_dir_exists [gw1] [ 4%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw9] [ 5%] PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 [gw4] [ 6%] PASSED test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options [gw9] [ 7%] PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 [gw1] [ 8%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] [gw3] [ 9%] PASSED test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts [gw2] [ 10%] PASSED test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy [gw6] [ 11%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys [gw7] [ 12%] PASSED test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials [gw1] [ 13%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw3] [ 14%] PASSED test_partition/test.py::test_drop_detached_parts [gw8] [ 15%] PASSED test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_partition/test.py::test_make_clone_in_detached [gw1] [ 16%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] [gw6] [ 17%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces [gw2] [ 18%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config [gw5] [ 19%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] [gw7] [ 20%] PASSED test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials [gw1] [ 21%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] [gw3] [ 22%] PASSED test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex [gw4] [ 23%] PASSED test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options [gw1] [ 24%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] [gw6] [ 25%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys [gw7] [ 26%] PASSED test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials [gw8] [ 27%] PASSED test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata [gw2] [ 28%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy [gw5] [ 29%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] [gw1] [ 30%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw3] [ 31%] PASSED test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple [gw3] [ 32%] PASSED test_partition/test.py::test_partition_simple [gw7] [ 33%] PASSED test_replication_credentials/test.py::test_same_credentials [gw1] [ 34%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] test_partition/test.py::test_system_detached_parts test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] [gw5] [ 35%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] [gw6] [ 36%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_same_keys test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards [gw1] [ 37%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] [gw2] [ 38%] PASSED test_reloading_storage_configuration/test.py::test_add_policy test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] test_reloading_storage_configuration/test.py::test_add_volume_to_policy [gw5] [ 39%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] [gw1] [ 40%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] [gw4] [ 41%] PASSED test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_replicated_table_attach/test.py::test_startup_with_small_bg_pool [gw1] [ 42%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] test_s3_style_link/test.py::test_s3_table_functions [gw2] [ 43%] PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works [gw1] [ 44%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw5] [ 45%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] [gw3] [ 46%] PASSED test_partition/test.py::test_system_detached_parts [gw2] [ 47%] PASSED test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk [gw7] [ 48%] PASSED test_s3_style_link/test.py::test_s3_table_functions test_s3_style_link/test.py::test_s3_table_functions_line_as_string [gw4] [ 49%] PASSED test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options [gw0] [ 50%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] [gw7] [ 51%] PASSED test_s3_style_link/test.py::test_s3_table_functions_line_as_string test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race [gw5] [ 52%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] test_s3_table_functions/test.py::test_s3_table_functions [gw0] [ 53%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] [gw2] [ 54%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy [gw8] [ 55%] PASSED test_recovery_replica/test.py::test_update_metadata [gw6] [ 56%] PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned [gw0] [ 57%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] test_part_uuid/test.py::test_part_uuid [gw4] [ 58%] PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options [gw0] [ 59%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] [gw2] [ 60%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reload_client_certificate/test.py::test_correct_cn_cert [gw9] [ 61%] PASSED test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards [gw0] [ 62%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] [gw3] [ 63%] PASSED test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts [gw2] [ 64%] PASSED test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy [gw0] [ 65%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] test_relative_filepath/test.py::test_filepath [gw5] [ 66%] PASSED test_part_uuid/test.py::test_part_uuid [gw4] [ 67%] PASSED test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options [gw4] [ 68%] PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options [gw4] [ 69%] PASSED test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options [gw0] [ 70%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge [gw4] [ 71%] PASSED test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options [gw6] [ 72%] PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned [gw2] [ 73%] PASSED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk [gw4] [ 74%] PASSED test_rocksdb_options/test.py::test_valid_options [gw7] [ 75%] PASSED test_relative_filepath/test.py::test_filepath [gw3] [ 76%] PASSED test_s3_table_functions/test.py::test_s3_table_functions_timeouts [gw1] [ 77%] FAILED test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_query_masking_rules/test.py::test_reload_query_masking_rules [gw2] [ 78%] PASSED test_reloading_storage_configuration/test.py::test_update_disk test_read_only_table/test.py::test_restart_zookeeper [gw0] [ 79%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter [gw6] [ 80%] PASSED test_range_hashed_dictionary_types/test.py::test_range_hashed_dict [gw7] [ 81%] PASSED test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_s3_imds/test_session_token.py::test_credentials_from_metadata [gw5] [ 82%] PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_replicating_constants/test.py::test_different_versions test_s3_imds/test_simple.py::test_credentials_from_metadata [gw0] [ 83%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system [gw9] [ 84%] PASSED test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards [gw5] [ 85%] PASSED test_s3_imds/test_simple.py::test_credentials_from_metadata test_parallel_replicas_increase_error_count/test.py::test_increase_error_count [gw6] [ 86%] PASSED test_s3_imds/test_session_token.py::test_credentials_from_metadata [gw3] [ 87%] PASSED test_replicating_constants/test.py::test_different_versions [gw8] [ 88%] PASSED test_reload_client_certificate/test.py::test_correct_cn_cert test_reload_client_certificate/test.py::test_wrong_cn_cert [gw0] [ 89%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] [gw1] [ 90%] PASSED test_read_only_table/test.py::test_restart_zookeeper test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach [gw3] [ 91%] PASSED test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] [gw9] [ 92%] PASSED test_parallel_replicas_increase_error_count/test.py::test_increase_error_count [gw0] [ 93%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter test_s3_storage_class/test.py::test_s3_storage_class_right [gw1] [ 94%] PASSED test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach [gw0] [ 95%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system [gw0] [ 96%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] [gw9] [ 97%] PASSED test_s3_storage_class/test.py::test_s3_storage_class_right [gw0] [ 98%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] [gw8] [ 99%] PASSED test_reload_client_certificate/test.py::test_wrong_cn_cert test_s3_latency_log/test.py::test_latency_log [gw8] [100%] PASSED test_s3_latency_log/test.py::test_latency_log =================================== FAILURES =================================== ________________________ test_partition_move_drop_race _________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_partition_move_drop_race(started_cluster): for node in [node1, node2, node3]: node.query(f"DROP DATABASE IF EXISTS test_db SYNC") node1.query("SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range") node1.query("SYSTEM ENABLE FAILPOINT rmt_delay_commit_part") for node in [node1, node2, node3]: node.query( """ CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') """ ) node1.query( f""" CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 """ ) for node in [node1, node2, node3]: assert node.query_with_retry( "SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl'", check_callback=lambda x: x.strip() == "1", ) table_uuid = node1.query( "SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl'" ).strip() node2.query(f"SYSTEM STOP FETCHES test_db.tbl") node3.query(f"SYSTEM STOP FETCHES test_db.tbl") prev_block_number = get_last_block_number_for_partition(node1, table_uuid, 0) exception_holder = [None] def drop_op(): try: # Wait until INSERT query allocates the block number timeout = 60 # seconds start_time = time.time() while True: current_block_number = get_last_block_number_for_partition( node1, table_uuid, 0 ) if current_block_number != prev_block_number: break if time.time() - start_time > timeout: raise TimeoutError( f"Timeout waiting for block number to change after {timeout}s. " f"Previous: {prev_block_number}, Current: {current_block_number}" ) time.sleep(0.5) node1.query(f"ALTER TABLE test_db.tbl DROP PARTITION 0") except Exception as e: exception_holder[0] = e t = threading.Thread(target=drop_op) t.start() > node1.query(f"INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c')") test_race_condition_for_replicated_merge_tree/test.py:146: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3649: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 138, stderr: Received exception from server (version 25.3.14): E Code: 394. DB::Exception: Received from 172.16.2.7:9000. DB::Exception: Insert query (for block /clickhouse/tables/fb7de1c3-0f33-4c38-9038-fa42666ff30b/s1/block_numbers/0/block-0000000000) was canceled by concurrent ALTER PARTITION or TRUNCATE. Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ecd1 E 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c089a11 E 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c420b E 3. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&&) @ 0x000000000c6ddf98 E 4. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1116: DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda1'()::operator()() const @ 0x000000002f9c44e3 E 5. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1146: DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda2'()::operator()() const @ 0x000000002f9c0d40 E 6. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1179: void DB::ZooKeeperRetriesControl::retryLoop::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&, void DB::ZooKeeperRetriesControl::retryLoop::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()>(DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&&)::'lambda'()>(DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&&, void DB::ZooKeeperRetriesControl::retryLoop::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()>(DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&&)::'lambda'()&&) @ 0x000000002f9c0855 E 7. ./src/Common/ZooKeeper/ZooKeeperRetries.h:52: DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long) @ 0x000000002f99d15d E 8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:493: DB::ReplicatedMergeTreeSinkImpl::finishDelayedChunk(std::shared_ptr const&) @ 0x000000002f983455 E 9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1257: DB::ReplicatedMergeTreeSinkImpl::onFinish() @ 0x000000002f99bb8c E 10. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000030905fbb E 11. ./build_docker/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:187: DB::ExceptionKeepingTransform::work() @ 0x00000000309053b7 E 12. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::ExecutionThreadContext::executeTask() @ 0x00000000302efa14 E 13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:296: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x00000000302d0560 E 14. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:170: DB::PipelineExecutor::executeStep(std::atomic*) @ 0x00000000302cf515 E 15. ./build_docker/./src/Processors/Executors/PushingPipelineExecutor.cpp:131: DB::PushingPipelineExecutor::finish() @ 0x0000000030300a55 E 16. ./build_docker/./src/Server/TCPHandler.cpp:1128: DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000300d9686 E 17. ./build_docker/./src/Server/TCPHandler.cpp:673: DB::TCPHandler::runImpl() @ 0x00000000300bf95a E 18. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3da E 19. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x0000000038859fcf E 20. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885ac97 E 21. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x00000000387695eb E 22. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000387631e8 E 23. asan_thread_start(void*) @ 0x000000000c676e77 E 24. ? @ 0x00007f9281a16ac3 E 25. ? @ 0x00007f9281aa8850 E . (QUERY_WAS_CANCELLED) E (query: INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c')) helpers/client.py:248: QueryRuntimeException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- Running tests in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/test.py Running tests in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/test.py Cluster start called. is_up=False Cluster start called. is_up=False Docker networks for project roottestraceconditionforreplicatedmergetree-gw1 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestraceconditionforreplicatedmergetree-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestraceconditionforreplicatedmergetree-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestraceconditionforreplicatedmergetree-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestraceconditionforreplicatedmergetree-gw1 are DRIVER VOLUME NAME Docker volumes for project roottestraceconditionforreplicatedmergetree-gw1 are DRIVER VOLUME NAME Cleanup called Cleanup called Docker networks for project roottestraceconditionforreplicatedmergetree-gw1 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestraceconditionforreplicatedmergetree-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestraceconditionforreplicatedmergetree-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestraceconditionforreplicatedmergetree-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestraceconditionforreplicatedmergetree-gw1 are DRIVER VOLUME NAME Docker volumes for project roottestraceconditionforreplicatedmergetree-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestraceconditionforreplicatedmergetree-gw1 No running containers for project: roottestraceconditionforreplicatedmergetree-gw1 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:5 Stdout:5 Command:[docker volume prune -f] Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Volumes pruned: 5 Volumes pruned: 5 Setup directory for instance: node1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/configs/config.d Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/database Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/database Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/logs Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup directory for instance: node2 Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/configs/config.d Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/database Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/database Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/logs Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup directory for instance: node3 Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/configs/config.d Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/configs/config.d Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/database Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/database Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/logs Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml pull] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml pull] Stderr: node1 Skipped - Image is already being pulled by node3 Stderr: node1 Skipped - Image is already being pulled by node3 Stderr: zoo1 Skipped - Image is already being pulled by node3 Stderr: zoo1 Skipped - Image is already being pulled by node3 Stderr: zoo2 Skipped - Image is already being pulled by node3 Stderr: zoo2 Skipped - Image is already being pulled by node3 Stderr: zoo3 Skipped - Image is already being pulled by node3 Stderr: zoo3 Skipped - Image is already being pulled by node3 Stderr: node2 Skipped - Image is already being pulled by node3 Stderr: node2 Skipped - Image is already being pulled by node3 Stderr: node3 Pulling Stderr: node3 Pulling Stderr: node3 Pulled Stderr: node3 Pulled Setup ZooKeeper Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/coordination'] Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/coordination'] Command:[docker compose --project-name roottestraceconditionforreplicatedmergetree-gw1 --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Command:[docker compose --project-name roottestraceconditionforreplicatedmergetree-gw1 --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr:time="2026-09-21T00:35:50Z" level=trace msg="Docker Desktop integration not enabled" Stderr:time="2026-09-21T00:35:50Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Creating Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Creating Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Created Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Started Stderr:time="2026-09-21T00:35:51Z" level=debug msg="otel error" error="" Stderr:time="2026-09-21T00:35:51Z" level=debug msg="otel error" error="" Stderr:time="2026-09-21T00:35:51Z" level=debug msg="otel error" error="" Stderr:time="2026-09-21T00:35:51Z" level=debug msg="otel error" error="" Wait ZooKeeper to start Wait ZooKeeper to start get_instance_ip instance_name=zoo1 get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-zoo1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.2.3, port:2181, use_ssl:False get_kazoo_client: zoo1, ip:172.16.2.3, port:2181, use_ssl:False Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-zoo2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.2.4, port:2181, use_ssl:False get_kazoo_client: zoo2, ip:172.16.2.4, port:2181, use_ssl:False Connecting to 172.16.2.4(172.16.2.4):2181, use_ssl: False Connecting to 172.16.2.4(172.16.2.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-zoo3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.2.2, port:2181, use_ssl:False get_kazoo_client: zoo3, ip:172.16.2.2, port:2181, use_ssl:False Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml up -d --no-recreate') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml up -d --no-recreate] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml up -d --no-recreate] Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Started ClickHouse instance created ClickHouse instance created get_instance_ip instance_name=node1 get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.2.7... Waiting for ClickHouse start in node1, ip: 172.16.2.7... http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None ClickHouse node1 started ClickHouse node1 started get_instance_ip instance_name=node2 get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.2.6... Waiting for ClickHouse start in node2, ip: 172.16.2.6... http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/72629b903c07bb4d6b3e7bcd713059696770e231aa47b47c0be1c77303aa6f2a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/72629b903c07bb4d6b3e7bcd713059696770e231aa47b47c0be1c77303aa6f2a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/72629b903c07bb4d6b3e7bcd713059696770e231aa47b47c0be1c77303aa6f2a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/72629b903c07bb4d6b3e7bcd713059696770e231aa47b47c0be1c77303aa6f2a/json HTTP/1.1" 200 None ClickHouse node2 started ClickHouse node2 started get_instance_ip instance_name=node3 get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node3 get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.2.5... Waiting for ClickHouse start in node3, ip: 172.16.2.5... http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/269faee3e381b2e529f279c73ad51acb77b71b885e46dd4333e9e612ca05f895/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/269faee3e381b2e529f279c73ad51acb77b71b885e46dd4333e9e612ca05f895/json HTTP/1.1" 200 None ClickHouse node3 started ClickHouse node3 started ------------------------------ Captured log setup ------------------------------ 2026-09-21 00:35:39.141000 [ 597 ] INFO : Running tests in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/test.py (cluster.py:2738, start) 2026-09-21 00:35:39.141000 [ 597 ] DEBUG : Cluster start called. is_up=False (cluster.py:2745, start) 2026-09-21 00:35:39.159000 [ 597 ] DEBUG : Docker networks for project roottestraceconditionforreplicatedmergetree-gw1 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 00:35:39.183000 [ 597 ] DEBUG : Docker containers for project roottestraceconditionforreplicatedmergetree-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 00:35:39.210000 [ 597 ] DEBUG : Docker volumes for project roottestraceconditionforreplicatedmergetree-gw1 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 00:35:39.210000 [ 597 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-09-21 00:35:39.227000 [ 597 ] DEBUG : Docker networks for project roottestraceconditionforreplicatedmergetree-gw1 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 00:35:39.249000 [ 597 ] DEBUG : Docker containers for project roottestraceconditionforreplicatedmergetree-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 00:35:39.270000 [ 597 ] DEBUG : Docker volumes for project roottestraceconditionforreplicatedmergetree-gw1 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 00:35:39.270000 [ 597 ] DEBUG : Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-09-21 00:35:39.293000 [ 597 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-09-21 00:35:39.294000 [ 597 ] DEBUG : No running containers for project: roottestraceconditionforreplicatedmergetree-gw1 (cluster.py:879, cleanup) 2026-09-21 00:35:39.294000 [ 597 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-09-21 00:35:39.315000 [ 597 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-09-21 00:35:39.315000 [ 597 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-09-21 00:35:39.339000 [ 597 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-09-21 00:35:39.340000 [ 597 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-09-21 00:35:39.340000 [ 597 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-09-21 00:35:39.340000 [ 597 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-09-21 00:35:39.362000 [ 597 ] DEBUG : Stdout:5 (cluster.py:145, run_and_check) 2026-09-21 00:35:39.362000 [ 597 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-09-21 00:35:39.379000 [ 597 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-09-21 00:35:39.379000 [ 597 ] DEBUG : Volumes pruned: 5 (cluster.py:915, cleanup) 2026-09-21 00:35:39.380000 [ 597 ] DEBUG : Setup directory for instance: node1 (cluster.py:2758, start) 2026-09-21 00:35:39.380000 [ 597 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-09-21 00:35:39.381000 [ 597 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-09-21 00:35:39.381000 [ 597 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-09-21 00:35:39.382000 [ 597 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-09-21 00:35:39.383000 [ 597 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/configs/config.d (cluster.py:4741, create_dir) 2026-09-21 00:35:39.384000 [ 597 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/database (cluster.py:4758, create_dir) 2026-09-21 00:35:39.384000 [ 597 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/logs (cluster.py:4769, create_dir) 2026-09-21 00:35:39.384000 [ 597 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4850, create_dir) 2026-09-21 00:35:39.384000 [ 597 ] DEBUG : Setup directory for instance: node2 (cluster.py:2758, start) 2026-09-21 00:35:39.384000 [ 597 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-09-21 00:35:39.385000 [ 597 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-09-21 00:35:39.385000 [ 597 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-09-21 00:35:39.385000 [ 597 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-09-21 00:35:39.385000 [ 597 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/configs/config.d (cluster.py:4741, create_dir) 2026-09-21 00:35:39.385000 [ 597 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/database (cluster.py:4758, create_dir) 2026-09-21 00:35:39.386000 [ 597 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/logs (cluster.py:4769, create_dir) 2026-09-21 00:35:39.386000 [ 597 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4850, create_dir) 2026-09-21 00:35:39.386000 [ 597 ] DEBUG : Setup directory for instance: node3 (cluster.py:2758, start) 2026-09-21 00:35:39.386000 [ 597 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-09-21 00:35:39.386000 [ 597 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-09-21 00:35:39.386000 [ 597 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-09-21 00:35:39.387000 [ 597 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-09-21 00:35:39.387000 [ 597 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/configs/config.d (cluster.py:4741, create_dir) 2026-09-21 00:35:39.387000 [ 597 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/database (cluster.py:4758, create_dir) 2026-09-21 00:35:39.387000 [ 597 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/logs (cluster.py:4769, create_dir) 2026-09-21 00:35:39.387000 [ 597 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4850, create_dir) 2026-09-21 00:35:39.387000 [ 597 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env (cluster.py:96, _create_env_file) 2026-09-21 00:35:39.390000 [ 597 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-09-21 00:35:39.391000 [ 597 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-09-21 00:35:39.391000 [ 597 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-09-21 00:35:39.391000 [ 597 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-09-21 00:35:39.409000 [ 597 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2026-09-21 00:35:39.411000 [ 597 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-09-21 00:35:49.937000 [ 597 ] DEBUG : Stderr: node1 Skipped - Image is already being pulled by node3 (cluster.py:147, run_and_check) 2026-09-21 00:35:49.937000 [ 597 ] DEBUG : Stderr: zoo1 Skipped - Image is already being pulled by node3 (cluster.py:147, run_and_check) 2026-09-21 00:35:49.937000 [ 597 ] DEBUG : Stderr: zoo2 Skipped - Image is already being pulled by node3 (cluster.py:147, run_and_check) 2026-09-21 00:35:49.937000 [ 597 ] DEBUG : Stderr: zoo3 Skipped - Image is already being pulled by node3 (cluster.py:147, run_and_check) 2026-09-21 00:35:49.937000 [ 597 ] DEBUG : Stderr: node2 Skipped - Image is already being pulled by node3 (cluster.py:147, run_and_check) 2026-09-21 00:35:49.937000 [ 597 ] DEBUG : Stderr: node3 Pulling (cluster.py:147, run_and_check) 2026-09-21 00:35:49.937000 [ 597 ] DEBUG : Stderr: node3 Pulled (cluster.py:147, run_and_check) 2026-09-21 00:35:49.937000 [ 597 ] DEBUG : Setup ZooKeeper (cluster.py:2799, start) 2026-09-21 00:35:49.938000 [ 597 ] DEBUG : Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper1/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper2/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/keeper3/coordination'] (cluster.py:2800, start) 2026-09-21 00:35:49.939000 [ 597 ] DEBUG : Command:[docker compose --project-name roottestraceconditionforreplicatedmergetree-gw1 --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] (cluster.py:121, run_and_check) 2026-09-21 00:35:51.326000 [ 597 ] DEBUG : Stderr:time="2026-09-21T00:35:50Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:147, run_and_check) 2026-09-21 00:35:51.326000 [ 597 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Creating (cluster.py:147, run_and_check) 2026-09-21 00:35:51.326000 [ 597 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Created (cluster.py:147, run_and_check) 2026-09-21 00:35:51.326000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Creating (cluster.py:147, run_and_check) 2026-09-21 00:35:51.326000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Creating (cluster.py:147, run_and_check) 2026-09-21 00:35:51.326000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Creating (cluster.py:147, run_and_check) 2026-09-21 00:35:51.327000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Created (cluster.py:147, run_and_check) 2026-09-21 00:35:51.327000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Created (cluster.py:147, run_and_check) 2026-09-21 00:35:51.327000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Created (cluster.py:147, run_and_check) 2026-09-21 00:35:51.327000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Starting (cluster.py:147, run_and_check) 2026-09-21 00:35:51.327000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Starting (cluster.py:147, run_and_check) 2026-09-21 00:35:51.327000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Starting (cluster.py:147, run_and_check) 2026-09-21 00:35:51.327000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Started (cluster.py:147, run_and_check) 2026-09-21 00:35:51.327000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Started (cluster.py:147, run_and_check) 2026-09-21 00:35:51.328000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Started (cluster.py:147, run_and_check) 2026-09-21 00:35:51.328000 [ 597 ] DEBUG : Stderr:time="2026-09-21T00:35:51Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-09-21 00:35:51.328000 [ 597 ] DEBUG : Stderr:time="2026-09-21T00:35:51Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-09-21 00:35:51.328000 [ 597 ] DEBUG : Wait ZooKeeper to start (cluster.py:2436, wait_zookeeper_to_start) 2026-09-21 00:35:51.328000 [ 597 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2005, get_instance_ip) 2026-09-21 00:35:51.332000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-zoo1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:51.333000 [ 597 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.2.3, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-09-21 00:35:51.341000 [ 597 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 00:35:51.342000 [ 597 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-09-21 00:35:51.422000 [ 597 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 00:35:51.422000 [ 597 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-09-21 00:35:51.628000 [ 597 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 00:35:51.629000 [ 597 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-09-21 00:35:51.972000 [ 597 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 00:35:51.973000 [ 597 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-09-21 00:35:52.588000 [ 597 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 00:35:52.589000 [ 597 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-09-21 00:35:53.612000 [ 597 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 00:35:53.612000 [ 597 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-09-21 00:35:56.306000 [ 597 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 00:35:56.307000 [ 597 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-09-21 00:35:56.312000 [ 597 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-09-21 00:35:56.314000 [ 597 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-09-21 00:35:56.315000 [ 597 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-09-21 00:35:56.315000 [ 597 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-09-21 00:35:56.319000 [ 597 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-09-21 00:35:56.319000 [ 597 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-09-21 00:35:56.319000 [ 597 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-09-21 00:35:56.386000 [ 597 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-09-21 00:35:56.387000 [ 597 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-09-21 00:35:56.387000 [ 597 ] DEBUG : get_instance_ip instance_name=zoo2 (cluster.py:2005, get_instance_ip) 2026-09-21 00:35:56.389000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-zoo2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:56.390000 [ 597 ] DEBUG : get_kazoo_client: zoo2, ip:172.16.2.4, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-09-21 00:35:56.391000 [ 597 ] INFO : Connecting to 172.16.2.4(172.16.2.4):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 00:35:56.391000 [ 597 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-09-21 00:35:56.395000 [ 597 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-09-21 00:35:56.395000 [ 597 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-09-21 00:35:56.396000 [ 597 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-09-21 00:35:56.396000 [ 597 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-09-21 00:35:56.399000 [ 597 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-09-21 00:35:56.399000 [ 597 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-09-21 00:35:56.399000 [ 597 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-09-21 00:35:56.499000 [ 597 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-09-21 00:35:56.500000 [ 597 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-09-21 00:35:56.500000 [ 597 ] DEBUG : get_instance_ip instance_name=zoo3 (cluster.py:2005, get_instance_ip) 2026-09-21 00:35:56.502000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-zoo3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:56.502000 [ 597 ] DEBUG : get_kazoo_client: zoo3, ip:172.16.2.2, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-09-21 00:35:56.503000 [ 597 ] INFO : Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False (connection.py:650, _connect) 2026-09-21 00:35:56.504000 [ 597 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-09-21 00:35:56.508000 [ 597 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-09-21 00:35:56.509000 [ 597 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-09-21 00:35:56.509000 [ 597 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-09-21 00:35:56.510000 [ 597 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-09-21 00:35:56.512000 [ 597 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-09-21 00:35:56.513000 [ 597 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-09-21 00:35:56.513000 [ 597 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-09-21 00:35:56.595000 [ 597 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-09-21 00:35:56.595000 [ 597 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-09-21 00:35:56.595000 [ 597 ] DEBUG : All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') (cluster.py:2452, wait_zookeeper_nodes_to_start) 2026-09-21 00:35:56.596000 [ 597 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml up -d --no-recreate') (cluster.py:3139, start) 2026-09-21 00:35:56.596000 [ 597 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml up -d --no-recreate] (cluster.py:121, run_and_check) 2026-09-21 00:35:57.225000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Running (cluster.py:147, run_and_check) 2026-09-21 00:35:57.225000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Running (cluster.py:147, run_and_check) 2026-09-21 00:35:57.225000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Running (cluster.py:147, run_and_check) 2026-09-21 00:35:57.225000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Creating (cluster.py:147, run_and_check) 2026-09-21 00:35:57.225000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Creating (cluster.py:147, run_and_check) 2026-09-21 00:35:57.225000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Creating (cluster.py:147, run_and_check) 2026-09-21 00:35:57.225000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Created (cluster.py:147, run_and_check) 2026-09-21 00:35:57.225000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Created (cluster.py:147, run_and_check) 2026-09-21 00:35:57.225000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Created (cluster.py:147, run_and_check) 2026-09-21 00:35:57.226000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Starting (cluster.py:147, run_and_check) 2026-09-21 00:35:57.226000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Starting (cluster.py:147, run_and_check) 2026-09-21 00:35:57.226000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Starting (cluster.py:147, run_and_check) 2026-09-21 00:35:57.226000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Started (cluster.py:147, run_and_check) 2026-09-21 00:35:57.226000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Started (cluster.py:147, run_and_check) 2026-09-21 00:35:57.226000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Started (cluster.py:147, run_and_check) 2026-09-21 00:35:57.226000 [ 597 ] DEBUG : ClickHouse instance created (cluster.py:3147, start) 2026-09-21 00:35:57.226000 [ 597 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2005, get_instance_ip) 2026-09-21 00:35:57.228000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:57.228000 [ 597 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2015, get_instance_global_ipv6) 2026-09-21 00:35:57.229000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:57.229000 [ 597 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.2.7... (cluster.py:3155, start) 2026-09-21 00:35:57.230000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:57.232000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:57.334000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:57.437000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:57.539000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:57.642000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:57.745000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:57.848000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:57.951000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.054000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.159000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.262000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.365000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.468000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.573000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/c3b07549436f62309731f4b13989974f078501ce300adf4707066643d56c2292/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.574000 [ 597 ] DEBUG : ClickHouse node1 started (cluster.py:3159, start) 2026-09-21 00:35:58.575000 [ 597 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2005, get_instance_ip) 2026-09-21 00:35:58.578000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.578000 [ 597 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2015, get_instance_global_ipv6) 2026-09-21 00:35:58.580000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.581000 [ 597 ] DEBUG : Waiting for ClickHouse start in node2, ip: 172.16.2.6... (cluster.py:3155, start) 2026-09-21 00:35:58.583000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.585000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/72629b903c07bb4d6b3e7bcd713059696770e231aa47b47c0be1c77303aa6f2a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.688000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/72629b903c07bb4d6b3e7bcd713059696770e231aa47b47c0be1c77303aa6f2a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.688000 [ 597 ] DEBUG : ClickHouse node2 started (cluster.py:3159, start) 2026-09-21 00:35:58.688000 [ 597 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2005, get_instance_ip) 2026-09-21 00:35:58.690000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.690000 [ 597 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2015, get_instance_global_ipv6) 2026-09-21 00:35:58.691000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.692000 [ 597 ] DEBUG : Waiting for ClickHouse start in node3, ip: 172.16.2.5... (cluster.py:3155, start) 2026-09-21 00:35:58.692000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw1-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.694000 [ 597 ] DEBUG : http://localhost:None "GET /v1.46/containers/269faee3e381b2e529f279c73ad51acb77b71b885e46dd4333e9e612ca05f895/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-09-21 00:35:58.695000 [ 597 ] DEBUG : ClickHouse node3 started (cluster.py:3159, start) ----------------------------- Captured stderr call ----------------------------- Executing query DROP DATABASE IF EXISTS test_db SYNC on node1 Executing query DROP DATABASE IF EXISTS test_db SYNC on node1 Executing query DROP DATABASE IF EXISTS test_db SYNC on node2 Executing query DROP DATABASE IF EXISTS test_db SYNC on node2 Executing query DROP DATABASE IF EXISTS test_db SYNC on node3 Executing query DROP DATABASE IF EXISTS test_db SYNC on node3 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range on node1 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range on node1 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_commit_part on node1 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_commit_part on node1 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node1 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node1 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node2 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node2 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node3 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node3 Executing query CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 on node1 Executing query CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 on node1 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node2 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node2 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node3 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node3 Executing query SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SYSTEM STOP FETCHES test_db.tbl on node2 Executing query SYSTEM STOP FETCHES test_db.tbl on node2 Executing query SYSTEM STOP FETCHES test_db.tbl on node3 Executing query SYSTEM STOP FETCHES test_db.tbl on node3 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fb7de1c3-0f33-4c38-9038-fa42666ff30b/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fb7de1c3-0f33-4c38-9038-fa42666ff30b/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fb7de1c3-0f33-4c38-9038-fa42666ff30b/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fb7de1c3-0f33-4c38-9038-fa42666ff30b/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c') on node1 Executing query INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c') on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fb7de1c3-0f33-4c38-9038-fa42666ff30b/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fb7de1c3-0f33-4c38-9038-fa42666ff30b/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query ALTER TABLE test_db.tbl DROP PARTITION 0 on node1 Executing query ALTER TABLE test_db.tbl DROP PARTITION 0 on node1 ------------------------------ Captured log call ------------------------------- 2026-09-21 00:35:58.698000 [ 597 ] DEBUG : Executing query DROP DATABASE IF EXISTS test_db SYNC on node1 (cluster.py:3648, query) 2026-09-21 00:35:58.913000 [ 597 ] DEBUG : Executing query DROP DATABASE IF EXISTS test_db SYNC on node2 (cluster.py:3648, query) 2026-09-21 00:35:59.128000 [ 597 ] DEBUG : Executing query DROP DATABASE IF EXISTS test_db SYNC on node3 (cluster.py:3648, query) 2026-09-21 00:35:59.395000 [ 597 ] DEBUG : Executing query SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range on node1 (cluster.py:3648, query) 2026-09-21 00:35:59.610000 [ 597 ] DEBUG : Executing query SYSTEM ENABLE FAILPOINT rmt_delay_commit_part on node1 (cluster.py:3648, query) 2026-09-21 00:35:59.926000 [ 597 ] DEBUG : Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node1 (cluster.py:3648, query) 2026-09-21 00:36:00.192000 [ 597 ] DEBUG : Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node2 (cluster.py:3648, query) 2026-09-21 00:36:00.408000 [ 597 ] DEBUG : Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node3 (cluster.py:3648, query) 2026-09-21 00:36:00.625000 [ 597 ] DEBUG : Executing query CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 on node1 (cluster.py:3648, query) 2026-09-21 00:36:00.991000 [ 597 ] DEBUG : Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node1 (cluster.py:3648, query) 2026-09-21 00:36:01.310000 [ 597 ] DEBUG : Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node2 (cluster.py:3648, query) 2026-09-21 00:36:01.626000 [ 597 ] DEBUG : Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node3 (cluster.py:3648, query) 2026-09-21 00:36:01.941000 [ 597 ] DEBUG : Executing query SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl' on node1 (cluster.py:3648, query) 2026-09-21 00:36:02.207000 [ 597 ] DEBUG : Executing query SYSTEM STOP FETCHES test_db.tbl on node2 (cluster.py:3648, query) 2026-09-21 00:36:02.424000 [ 597 ] DEBUG : Executing query SYSTEM STOP FETCHES test_db.tbl on node3 (cluster.py:3648, query) 2026-09-21 00:36:02.640000 [ 597 ] DEBUG : Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fb7de1c3-0f33-4c38-9038-fa42666ff30b/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 (cluster.py:3648, query) 2026-09-21 00:36:02.910000 [ 597 ] DEBUG : Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fb7de1c3-0f33-4c38-9038-fa42666ff30b/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 (cluster.py:3648, query) 2026-09-21 00:36:02.910000 [ 597 ] DEBUG : Executing query INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c') on node1 (cluster.py:3648, query) 2026-09-21 00:36:03.678000 [ 597 ] DEBUG : Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fb7de1c3-0f33-4c38-9038-fa42666ff30b/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 (cluster.py:3648, query) 2026-09-21 00:36:03.896000 [ 597 ] DEBUG : Executing query ALTER TABLE test_db.tbl DROP PARTITION 0 on node1 (cluster.py:3648, query) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml stop --timeout 20] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml stop --timeout 20] Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml down --volumes] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml down --volumes] Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Removed Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Removing Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Removing Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Removed Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Removed Cleanup called Cleanup called Docker networks for project roottestraceconditionforreplicatedmergetree-gw1 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestraceconditionforreplicatedmergetree-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestraceconditionforreplicatedmergetree-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestraceconditionforreplicatedmergetree-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestraceconditionforreplicatedmergetree-gw1 are DRIVER VOLUME NAME Docker volumes for project roottestraceconditionforreplicatedmergetree-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestraceconditionforreplicatedmergetree-gw1 No running containers for project: roottestraceconditionforreplicatedmergetree-gw1 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:8 Stdout:8 Command:[docker volume prune -f] Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Volumes pruned: 8 Volumes pruned: 8 ---------------------------- Captured log teardown ----------------------------- 2026-09-21 00:36:08.576000 [ 597 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml stop --timeout 20] (cluster.py:121, run_and_check) 2026-09-21 00:36:11.047000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:11.048000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:11.048000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:11.048000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:11.048000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:11.048000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:11.048000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:11.048000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:11.049000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:11.049000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:11.049000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:11.049000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:11.049000 [ 597 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-09-21 00:36:11.066000 [ 597 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-09-21 00:36:11.082000 [ 597 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-09-21 00:36:11.100000 [ 597 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/.env --project-name roottestraceconditionforreplicatedmergetree-gw1 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw1/node3/docker-compose.yml down --volumes] (cluster.py:121, run_and_check) 2026-09-21 00:36:12.952000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:12.952000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:12.952000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:12.952000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:12.952000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Removing (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Removing (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Removing (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node3-1 Removed (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node2-1 Removed (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-node1-1 Removed (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Removing (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Removing (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Removing (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo3-1 Removed (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo1-1 Removed (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw1-zoo2-1 Removed (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Removing (cluster.py:147, run_and_check) 2026-09-21 00:36:12.953000 [ 597 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw1_default Removed (cluster.py:147, run_and_check) 2026-09-21 00:36:12.954000 [ 597 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-09-21 00:36:12.979000 [ 597 ] DEBUG : Docker networks for project roottestraceconditionforreplicatedmergetree-gw1 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 00:36:12.999000 [ 597 ] DEBUG : Docker containers for project roottestraceconditionforreplicatedmergetree-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 00:36:13.025000 [ 597 ] DEBUG : Docker volumes for project roottestraceconditionforreplicatedmergetree-gw1 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 00:36:13.026000 [ 597 ] DEBUG : Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-09-21 00:36:13.046000 [ 597 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-09-21 00:36:13.046000 [ 597 ] DEBUG : No running containers for project: roottestraceconditionforreplicatedmergetree-gw1 (cluster.py:879, cleanup) 2026-09-21 00:36:13.046000 [ 597 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-09-21 00:36:13.067000 [ 597 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-09-21 00:36:13.067000 [ 597 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-09-21 00:36:13.087000 [ 597 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-09-21 00:36:13.088000 [ 597 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-09-21 00:36:13.088000 [ 597 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-09-21 00:36:13.088000 [ 597 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-09-21 00:36:13.110000 [ 597 ] DEBUG : Stdout:8 (cluster.py:145, run_and_check) 2026-09-21 00:36:13.111000 [ 597 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-09-21 00:36:13.137000 [ 597 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-09-21 00:36:13.137000 [ 597 ] DEBUG : Volumes pruned: 8 (cluster.py:915, cleanup) =============================== warnings summary =============================== test_reload_client_certificate/test.py: 45 warnings /usr/local/lib/python3.10/dist-packages/kazoo/handlers/utils.py:225: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ----------------- generated report log file: parallel2_0.jsonl ----------------- ============================== slowest durations =============================== 184.62s call test_reload_client_certificate/test.py::test_wrong_cn_cert 43.10s call test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards 41.70s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] 37.84s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] 37.28s call test_reload_client_certificate/test.py::test_correct_cn_cert 33.06s call test_recovery_replica/test.py::test_update_metadata 31.16s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 27.74s setup test_read_only_table/test.py::test_restart_zookeeper 25.65s setup test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 25.19s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] 24.50s teardown test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 23.71s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] 22.57s teardown test_s3_imds/test_session_token.py::test_credentials_from_metadata 22.25s teardown test_s3_imds/test_simple.py::test_credentials_from_metadata 22.16s teardown test_s3_style_link/test.py::test_s3_table_functions_line_as_string 22.00s setup test_replication_credentials/test.py::test_credentials_and_no_credentials 21.93s teardown test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] 21.91s teardown test_s3_table_functions/test.py::test_s3_table_functions_timeouts 21.89s teardown test_s3_storage_class/test.py::test_s3_storage_class_right 21.86s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] 21.57s teardown test_s3_latency_log/test.py::test_latency_log 21.36s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 20.82s setup test_recovery_replica/test.py::test_choose_source_replica 20.17s call test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 19.74s setup test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards 19.63s setup test_reloading_storage_configuration/test.py::test_add_disk 19.56s setup test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race 19.55s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter 19.46s setup test_reload_client_certificate/test.py::test_correct_cn_cert 19.19s setup test_replicating_constants/test.py::test_different_versions 19.06s call test_read_only_table/test.py::test_restart_zookeeper 18.83s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] 18.81s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system 18.51s setup test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 17.84s call test_partition/test.py::test_system_detached_parts 17.75s setup test_s3_low_cardinality_right_border/test.py::test_s3_right_border 17.72s call test_s3_latency_log/test.py::test_latency_log 17.18s setup test_s3_imds/test_session_token.py::test_credentials_from_metadata 17.01s teardown test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 16.44s setup test_s3_latency_log/test.py::test_latency_log 15.54s setup test_s3_storage_class/test.py::test_s3_storage_class_right 15.46s call test_rocksdb_options/test.py::test_invalid_options 15.30s setup test_partition/test.py::test_attach_check_all_parts 14.63s call test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards 14.09s call test_rocksdb_options/test.py::test_invalid_column_family_options 14.05s call test_rocksdb_options/test.py::test_table_invalid_options 13.33s setup test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 13.23s setup test_s3_table_functions/test.py::test_s3_table_functions 13.01s setup test_rocksdb_options/test.py::test_invalid_block_based_table_options 12.81s setup test_s3_imds/test_simple.py::test_credentials_from_metadata 12.50s call test_s3_table_functions/test.py::test_s3_table_functions_timeouts 12.48s call test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 11.81s teardown test_replication_credentials/test.py::test_same_credentials 11.75s setup test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 11.65s call test_rocksdb_options/test.py::test_table_invalid_column_family_options 11.44s call test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] 9.94s setup test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 9.93s call test_reloading_storage_configuration/test.py::test_add_policy 9.88s call test_rocksdb_options/test.py::test_invalid_block_based_table_options 9.88s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge 9.78s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter 9.65s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 9.57s setup test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 9.51s call test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race 9.21s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] 9.13s call test_recovery_replica/test.py::test_choose_source_replica 9.12s setup test_part_uuid/test.py::test_part_uuid 9.04s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 9.02s teardown test_recovery_replica/test.py::test_update_metadata 9.00s setup test_s3_style_link/test.py::test_s3_table_functions 8.59s call test_recovery_replica/test.py::test_recovery 8.55s call test_reloading_storage_configuration/test.py::test_new_policy_works 8.53s teardown test_part_uuid/test.py::test_part_uuid 8.38s teardown test_read_only_table/test.py::test_restart_zookeeper 8.09s teardown test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 7.94s call test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 7.89s teardown test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards 7.84s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] 7.82s teardown test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 7.76s call test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 7.63s call test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 7.51s teardown test_reload_client_certificate/test.py::test_wrong_cn_cert 7.42s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] 7.28s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] 7.26s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] 7.10s call test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 6.99s call test_reloading_storage_configuration/test.py::test_add_volume_to_policy 6.93s call test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 6.92s setup test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] 6.77s call test_reloading_storage_configuration/test.py::test_update_disk 6.72s call test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 6.66s call test_reloading_storage_configuration/test.py::test_remove_disk 6.64s call test_reloading_storage_configuration/test.py::test_remove_policy 6.54s call test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 6.34s call test_partition/test.py::test_make_clone_in_detached 6.31s call test_reloading_storage_configuration/test.py::test_add_disk_to_policy 6.12s call test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 6.04s setup test_relative_filepath/test.py::test_filepath 5.94s call test_replication_credentials/test.py::test_different_credentials 5.92s call test_part_uuid/test.py::test_part_uuid 5.81s call test_reloading_storage_configuration/test.py::test_add_disk 5.50s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] 5.49s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] 5.48s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] 5.21s teardown test_replicating_constants/test.py::test_different_versions 5.20s teardown test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 5.16s call test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 4.92s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] 4.83s teardown test_relative_filepath/test.py::test_filepath 4.67s call test_replication_credentials/test.py::test_credentials_and_no_credentials 4.63s teardown test_partition/test.py::test_system_detached_parts 4.60s call test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 4.56s teardown test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race 4.44s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] 4.37s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] 4.21s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] 4.19s teardown test_reloading_storage_configuration/test.py::test_update_disk 4.16s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] 4.09s call test_replication_credentials/test.py::test_no_credentials 4.06s teardown test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 3.88s call test_partition/test.py::test_drop_detached_parts 3.84s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 3.80s teardown test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 3.77s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] 3.62s call test_replication_credentials/test.py::test_same_credentials 3.50s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] 3.42s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 3.41s call test_partition/test.py::test_detached_part_dir_exists 3.33s call test_s3_low_cardinality_right_border/test.py::test_s3_right_border 3.17s setup test_partition/test.py::test_system_detached_parts 3.03s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] 3.00s teardown test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 2.98s call test_partition/test.py::test_attach_check_all_parts 2.90s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 2.85s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 2.79s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 2.57s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 2.56s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] 2.56s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] 2.54s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 2.31s call test_partition/test.py::test_partition_complex 1.89s call test_rocksdb_options/test.py::test_valid_options 1.73s call test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 1.73s call test_s3_imds/test_session_token.py::test_credentials_from_metadata 1.70s call test_s3_imds/test_simple.py::test_credentials_from_metadata 1.55s teardown test_rocksdb_options/test.py::test_valid_options 1.40s setup test_partition/test.py::test_drop_detached_parts 1.36s call test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 1.26s setup test_partition/test.py::test_partition_complex 1.18s setup test_partition/test.py::test_partition_simple 1.14s setup test_partition/test.py::test_cannot_attach_active_part 1.02s call test_s3_style_link/test.py::test_s3_table_functions_line_as_string 0.97s call test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 0.96s setup test_replication_credentials/test.py::test_no_credentials 0.94s setup test_replication_credentials/test.py::test_same_credentials 0.92s call test_partition/test.py::test_partition_simple 0.91s call test_partition/test.py::test_cannot_attach_active_part 0.90s call test_s3_storage_class/test.py::test_s3_storage_class_right 0.89s call test_s3_style_link/test.py::test_s3_table_functions 0.82s call test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 0.78s call test_s3_table_functions/test.py::test_s3_table_functions 0.77s setup test_replication_credentials/test.py::test_different_credentials 0.76s call test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 0.68s teardown test_partition/test.py::test_drop_detached_parts 0.66s call test_relative_filepath/test.py::test_filepath 0.60s teardown test_partition/test.py::test_attach_check_all_parts 0.32s call test_replicating_constants/test.py::test_different_versions 0.28s teardown test_partition/test.py::test_cannot_attach_active_part 0.27s call test_rocksdb_options/test.py::test_table_valid_options 0.27s call test_rocksdb_options/test.py::test_valid_column_family_options 0.27s teardown test_partition/test.py::test_partition_simple 0.27s call test_rocksdb_options/test.py::test_table_valid_column_family_options 0.23s teardown test_partition/test.py::test_partition_complex 0.01s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] 0.01s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s setup test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s teardown test_reload_client_certificate/test.py::test_correct_cn_cert 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 0.00s teardown test_partition/test.py::test_make_clone_in_detached 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_policy 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] 0.00s setup test_partition/test.py::test_make_clone_in_detached 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s setup test_rocksdb_options/test.py::test_invalid_options 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] 0.00s teardown test_replication_credentials/test.py::test_different_credentials 0.00s teardown test_rocksdb_options/test.py::test_invalid_column_family_options 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s setup test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s teardown test_rocksdb_options/test.py::test_valid_column_family_options 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s setup test_rocksdb_options/test.py::test_valid_options 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter 0.00s setup test_s3_table_functions/test.py::test_s3_table_functions_timeouts 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] 0.00s setup test_rocksdb_options/test.py::test_valid_column_family_options 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 0.00s teardown test_replication_credentials/test.py::test_credentials_and_no_credentials 0.00s setup test_rocksdb_options/test.py::test_table_valid_options 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter 0.00s teardown test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards 0.00s setup test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards 0.00s teardown test_rocksdb_options/test.py::test_table_valid_options 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 0.00s teardown test_recovery_replica/test.py::test_choose_source_replica 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 0.00s setup test_reloading_storage_configuration/test.py::test_update_disk 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_options 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 0.00s setup test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s teardown test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s setup test_reload_client_certificate/test.py::test_wrong_cn_cert 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] 0.00s teardown test_s3_table_functions/test.py::test_s3_table_functions 0.00s setup test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk 0.00s setup test_rocksdb_options/test.py::test_invalid_column_family_options 0.00s teardown test_reloading_storage_configuration/test.py::test_add_policy 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 0.00s setup test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 0.00s teardown test_s3_style_link/test.py::test_s3_table_functions 0.00s setup test_s3_style_link/test.py::test_s3_table_functions_line_as_string 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] 0.00s setup test_partition/test.py::test_detached_part_dir_exists 0.00s setup test_rocksdb_options/test.py::test_table_invalid_options 0.00s teardown test_replication_credentials/test.py::test_no_credentials 0.00s teardown test_rocksdb_options/test.py::test_invalid_options 0.00s setup test_reloading_storage_configuration/test.py::test_remove_policy 0.00s setup test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s setup test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_rocksdb_options/test.py::test_invalid_block_based_table_options 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s setup test_reloading_storage_configuration/test.py::test_add_policy 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] 0.00s setup test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 0.00s setup test_reloading_storage_configuration/test.py::test_new_policy_works 0.00s teardown test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 0.00s setup test_recovery_replica/test.py::test_recovery 0.00s teardown test_reloading_storage_configuration/test.py::test_new_policy_works 0.00s setup test_reloading_storage_configuration/test.py::test_remove_disk 0.00s teardown test_s3_low_cardinality_right_border/test.py::test_s3_right_border 0.00s setup test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 0.00s setup test_recovery_replica/test.py::test_update_metadata 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.00s teardown test_partition/test.py::test_detached_part_dir_exists 0.00s teardown test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s teardown test_recovery_replica/test.py::test_recovery 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] =========================== short test summary info ============================ FAILED test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race PASSED test_partition/test.py::test_attach_check_all_parts PASSED test_partition/test.py::test_cannot_attach_active_part PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 PASSED test_rocksdb_options/test.py::test_invalid_block_based_table_options PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_partition/test.py::test_detached_part_dir_exists PASSED test_reloading_storage_configuration/test.py::test_add_disk PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids PASSED test_replication_credentials/test.py::test_credentials_and_no_credentials PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] PASSED test_partition/test.py::test_drop_detached_parts PASSED test_recovery_replica/test.py::test_choose_source_replica PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_keys PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] PASSED test_replication_credentials/test.py::test_different_credentials PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] PASSED test_partition/test.py::test_make_clone_in_detached PASSED test_rocksdb_options/test.py::test_invalid_column_family_options PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces PASSED test_replication_credentials/test.py::test_no_credentials PASSED test_recovery_replica/test.py::test_recovery PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] PASSED test_partition/test.py::test_partition_complex PASSED test_partition/test.py::test_partition_simple PASSED test_replication_credentials/test.py::test_same_credentials PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_same_keys PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] PASSED test_reloading_storage_configuration/test.py::test_add_policy PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] PASSED test_rocksdb_options/test.py::test_invalid_options PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] PASSED test_partition/test.py::test_system_detached_parts PASSED test_reloading_storage_configuration/test.py::test_new_policy_works PASSED test_s3_style_link/test.py::test_s3_table_functions PASSED test_rocksdb_options/test.py::test_table_invalid_block_based_table_options PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] PASSED test_s3_style_link/test.py::test_s3_table_functions_line_as_string PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] PASSED test_reloading_storage_configuration/test.py::test_remove_disk PASSED test_recovery_replica/test.py::test_update_metadata PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy PASSED test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] PASSED test_s3_table_functions/test.py::test_s3_table_functions PASSED test_reloading_storage_configuration/test.py::test_remove_policy PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] PASSED test_part_uuid/test.py::test_part_uuid PASSED test_rocksdb_options/test.py::test_table_invalid_options PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options PASSED test_rocksdb_options/test.py::test_table_valid_options PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] PASSED test_rocksdb_options/test.py::test_valid_column_family_options PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned PASSED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy PASSED test_rocksdb_options/test.py::test_valid_options PASSED test_relative_filepath/test.py::test_filepath PASSED test_s3_table_functions/test.py::test_s3_table_functions_timeouts PASSED test_reloading_storage_configuration/test.py::test_update_disk PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge PASSED test_range_hashed_dictionary_types/test.py::test_range_hashed_dict PASSED test_reload_query_masking_rules/test.py::test_reload_query_masking_rules PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter PASSED test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards PASSED test_s3_imds/test_simple.py::test_credentials_from_metadata PASSED test_s3_imds/test_session_token.py::test_credentials_from_metadata PASSED test_replicating_constants/test.py::test_different_versions PASSED test_reload_client_certificate/test.py::test_correct_cn_cert PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system PASSED test_read_only_table/test.py::test_restart_zookeeper PASSED test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] PASSED test_parallel_replicas_increase_error_count/test.py::test_increase_error_count PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation PASSED test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system PASSED test_s3_storage_class/test.py::test_s3_storage_class_right PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] PASSED test_reload_client_certificate/test.py::test_wrong_cn_cert PASSED test_s3_latency_log/test.py::test_latency_log ============ 1 failed, 99 passed, 45 warnings in 387.43s (0:06:27) ============= Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 492, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_nsjvi9 --privileged --dns-search='.' --memory=30703247360 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_OLD_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel2_0.jsonl --report-log-exclude-logs-on-passed-tests 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1]' test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards test_part_uuid/test.py::test_part_uuid test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple test_partition/test.py::test_system_detached_parts test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_relative_filepath/test.py::test_filepath test_reload_client_certificate/test.py::test_correct_cn_cert test_reload_client_certificate/test.py::test_wrong_cn_cert test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_s3_imds/test_session_token.py::test_credentials_from_metadata test_s3_imds/test_simple.py::test_credentials_from_metadata test_s3_latency_log/test.py::test_latency_log test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 test_s3_storage_class/test.py::test_s3_storage_class_right 'test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3]' test_s3_style_link/test.py::test_s3_table_functions test_s3_style_link/test.py::test_s3_table_functions_line_as_string test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 ' returned non-zero exit status 1.